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Abstract 

We discuss, in this paper, the dynamical properties of extremely 
diluted, non-monotonic neural networks. Assuming parallel updat- 
ing and the Hebb prescription for the synaptic connections, a flow 
equation for the macroscopic overlap is derived. A rich dynamical 
phase diagram, was obtained, showing a stable retrieval phase, as well 
as a cycle two and chaotic behavior. Numerical simulations were per- 
formed, showing good agreement with analytical results. Furthermore, 
the simulations give an additional insight into the microscopic dynam- 
ical behavior during the chaotic phase. It is shown that the freezing 
of individual neuron states is related to the structure of chaotic at- 
tractors. 



1. Introduction 



Models of neural networks have been largely studied since the pioneering 
work of Hopfield Q. Assuming symmetric interactions and monotonic units, 
and using classical tools of statistical mechanics, equilibrium properties were 
extensively investigated. For a review in this subject, see, e. g., @]. Never- 
theless, the brain is a highly dynamical system, and equilibrium statistical 
mechanics should be not able to describe most of its properties. At our 
knowledge, the first work including the dynamical behavior as an important 
topic in neural networks research is due to Sompolinsky et al. ||. In that 
work, it was shown that asymmetric synaptic connections between units, a 
plausible feature, results in cyclic and chaotic behavior in the network dynam- 
ics. The chaotic behavior in neural tissue is, in fact supported by biological 
observations j|. 

More recently, a new class of models allowing for the complex behavior 
in neural networks started to be explored. It consists in networks of neurons 
in which the output activity is a non-monotonic function of the local input 
field. Next, we present a quick list with some important results available in 
the literature. 

The feed-forward perceptron is an architecture where the main task to be 
performed is the classification of a set of input patterns. Following first step 
replica symmetry breaking calculations in the space of the synaptic interac- 
tions, see [|, || and references therein, the optimal capacity for classification 
of patterns is larger for the non-monotonic than for the monotonic percep- 
tron. Still concerning this subject, non-monotonic perceptrons offer a further 
opportunity to study the fractal organization of the space of interactions fTj . 

In networks of recurrent architecture, the main subject concerns the asso- 
ciative memory property. If the dynamical evolution is governed by an hamil- 
tonian, with the relaxation to an equilibrium distribution of states reproduc- 
ing the process of memory retrieval. The question here is, if the dynamics 
is not governed by an hamiltonian, as in the case of non-monotonic, under 
which conditions it evolves to an equilibrium distribution of states keeping, 
in this way, the associative memory property? This problem was studied by 
Inoue ||, through the use of an equilibrium signal to noise analysis. It was 
found that, depending on the parameters defining the non-monotonicity, for 
binary patterns, and using the Hebb learning rule, the storage capacity, to 
be defined and discussed later, is highly improved, compared to monotonic 
neurons. 
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The dynamics of extremely diluted networks of continuous, non-monotonic 
neurons was studied in ||. The associative memory property is still present, 
and a non-regular, chaotic-type attractor was also noticed. The authors 
pointed out the emergence of chaos as a consequence of the non-monotonicity. 
The categorization ability of extremely diluted, three-state, non-monotonic 
neural network was studied in JTJ, |TjJ. A categorization phase, as well as a 
chaotic phase, was noticed. In jy] there is a study of the chaotic attractors 
of a model with non-monotonic, binary neurons, with finite connectivity and 
binary synapses. 

Our purpose in this paper is to study some of the dynamical properties 
of non-monotonic neural networks. The relaxation to the retrieval state, 
the appearing of oscillations and chaos, and the interplay between all these 
behaviors are discussed. In order to make transparent the essential physical 
aspects, the simplest model of non-monotonic neural network was chosen, i. 
e., the network of binary, reverse- wedge neurons, with hebbian connections. 
In order to have an analytically solvable dynamics, we restrict our study to 
the extremely diluted architecture with parallel updates. In addition, further 
insight on the microscopic dynamics during the chaotic phase, as well as in 
the transition from the chaotic to the retrieval phase, was obtained through 
numerical simulations. 



2. Diluted network dynamics 

We consider a network composed by N sites, whose state is represented by a 
set of variables {Si} i = 1 . . . N, with the state of the neuron in site i being 
represented by a binary variable Si — ±1. In the absence of retrieval noise, 
each neuron updates in a synchronous, deterministic way according to 

3i(t + l)=F e (hi(t)) , (1) 

where 

p (h) - { +1 : h < -8 and < h < 6 , . 

1—1 : otherwise 

is the reverse-wedge activation function. Here, 9 is a threshold parameter. 
The local field felt by the neuron at site % is given by 

N 

//,(/) »,' s ',- (3) 

i=i 
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In the limit of extreme dilution each neuron is connected to C other neurons, 
with 

1<C< In N. (4) 

Due to this condition, the synaptic connections have a tree-like structure, 
and the existence of feedback- loops has a vanishing probability ||. In con- 
sequence, the dynamics of this network is exactly solvable The storage 
capacity of the network is defined as a = p/C. We assume that the synapses 
are given by a Hebb-like learning rule, 

J* = 7? t , (5) 

where CV,- are random binary variables assuming values 1, if the output of 
neuron j is connected to the input of neuron i and zero, otherwise. They are 
chosen according the probability distribution 

p(Cij) = ^m 3 - 1) + (i - §) s(c i:i ) , (6) 

where S(x) is the Dirac's delta function. The memories {£f } , i — 1 . . . N , fi 
1 . . .p are a set of independent, identically distributed random binary vari- 
ables, chosen according the probability distribution 

^(£f) = ^£f-i) + ^(£f + i) • (7) 

The only relevant order parameters describing the macroscopic state of the 
network is the set of overlaps 

"V = ^E^ (8) 
JV i=l 

between the system configuration the stored patterns. 

In what follows, we consider that only one memory pattern, say fi = 1, 
has a macroscopic overlap of order 0(1), and all the others p—1 are of order 
0(1/ 'VN). Following the now standard derivation of |l2 |, we obtain that, 



in the absence of retrieval noise, the retrieval overlap satisfies the recursion 
relation 

mi(i + 1) = _I erf (WQ+fl + if (^g] _ I erf (^t)-e\ ^ 



2 \ J2a 
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For parallel dynamics, in the limit of extreme dilution, this equation fully 
describes the dynamical evolution of the network. 

Before to extend the discussion for all 9, it worths to mention what is 
the behavior of the system in the limits 9 — > oo and 9 — > 0. For 9 — ► 
oo, the deterministic neuron update reduces to the signal transfer function 
Si(t + 1) = sgn[hi(t)]. The macroscopic dynamical evolution is given by 
m(t + l) = erf(m(t)/v / 2a)- According to fll2"f , this equation has a fixed-point 
for t — > oo at a finite value m = m(a), provided that a <2/ix. At this value 
of the storage capacity, the system undergoes a continuous transition to the 
fixed point m — 0. 

In the limit 9 — > 0, the neurons update according to the transfer function 
Si(t + 1) = —sga[hi(t)], with the macroscopic dynamics being given by m(t + 
1) = — exl{m{t) / y2a) . This equation no longer have a finite fixed-point for 
finite m. Now, for small a, the network evolves dynamically through a cycle 
of length two, where the finite retrieval overlap changes from m to — m from 
one time step to the other. In other words, in one time step, it retrieves one 
pattern, and in the next time step it retrieves the inverse of this pattern. The 
amplitude m of the limit cycle decreases continuously with the increasing a, 
arriving to m = at the same storage capacity a = 2/tt as in the previous 
case 9 — > oo. 

Now we start to discuss the long term behavior of the system. A complete 
dynamical phase diagram for general 9 is shown in figure 1. In region I, 
corresponding to large a, the only fixed point of Eq. (Rf) is m = 0, known 



as the zero solution ||10|| . The region II, for intermediate and large 9, is the 
locus of the retrieval solution, with finite fixed point overlap m ^ 0. The 
transition between regions I and II is continuous. It worths to remark that 
the borderline for the retrieval region decreases monotonically from a = 2/tt 
for large 9 to a = for 9 = 0. This is in contrast to that reported by Inoue M 
for the fully connected network. There, the Hopfield limit a ~ 0.138 is found 
for large 9. Then, decreasing 9, the borderline for the retrieval increases up 
to a maximum a ~ 0.211 at 9 ~ 1.77. For smaller 9, it decreases till to reach 
a = at 9 = 0. 

The region III is divided in two parts, one for small 9 and a < 2/tt and 
other close to the the 9 axis, for 9 < 1. In this region, the network shows the 
cycle two behavior, as in the 9 — > limit. The amplitude of the cycle two 
decreases monotonically to zero at the transition line between regions I and 
III. The transition from region III to other region than I is discontinuous. 

In region IV the network has a periodic behavior. Leaving the retrieval re- 



4 



gion, roughly at the right and above, in the figure, it starts a period doubling 
cascade that ends in the chaotic region V. 

Finally, we discuss the regions that are signaled with a star in figure 1. 
The star means that the long term behavior depends on the initial value mo of 
the overlap. If the initial value is sufficiently small, the network evolves to the 
basin of attraction of the corresponding phase without the star. Otherwise, 
it ends in the cycle two behavior, like in phase III. 



3. Numerical simulations 

The non-monotonic network with extremely diluted connections offers an 
interesting opportunity to compare microscopic and macroscopic views of 
non-linear phenomena. As an example, the bifurcation diagram of the over- 
lap m for a = 0.04 and varying 9 is shown in figures 2a and 2b, determined, 
respectively from the macroscopic map of Eq. @ and the microscopic defi- 
nition of Eq. ([8]). The initial condition is m = 0.1, i. e., outside the basin of 
attraction of the cycle two solution. Figure 2b results from simulation over 
a network with N = 10000 and C = 100. In order to save computer mem- 
ory, only nonzero elements of the synaptic matrix where stored, as in Jl3| 



Comparing the two diagrams, we can see that bifurcations and periodicity 
windows are less clearly defined in the simulation than in the macroscopic 
evolution. Furthermore, in the range in 9 where states with m > and m < 
coexist in the same attractor, the diagram from simulation is more populated 
with states with small overlap. This is in contrast with the macroscopic evo- 
lution, that results in a more uniformly distributed diagram. We may ask if 
these discrepancies are due to the high connectivity value, since C/N = 0.01 
in figure 2b, while figure 2a holds in the limit C/N — > 0. We believe that this 
is not the case, since simulations with iV = 1000 and C = 100 give similar 
results. Instead of this, we believe that the discrepancies are due to finite size 
effects, since the definition a = p/C holds for both p and C going to oo while, 
in the simulations we have p = 4. Nevertheless, although the discrepancies, 
there is a good general agreement between the two diagrams. In particular, 
the figures agree in which concerns to transitions from cyclic to chaos and 
to retrieval regions. This means that, even if Eq. ([|) is not strictly satisfied, 
the simulations reproduce quite well the dynamics of the diluted network. 
With this, we conclude that it is justified to proceed with simulations, in 
order to obtain additional insight about the chaotic behavior of the network. 
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For completeness it is shown, in figure 2c, the Lyapunov exponent calculated 
from iterations of the macroscopic map (Eq||). The positive values assumed 
confirm that the system displays a chaotic phase. 

The description of the dynamical evolution through macroscopic variables 
leaves open some questions, mainly concerning the chaotic regime. We could 
ask, for instance, if the abrupt change in the overlap from step to step results 
from alternate flipping activity in all network sites, or if part of the sites are 
frozen and part are flipping. In order to have some insight about this and 
other questions, computational simulations were realized. Here we introduce, 
for each neuron i, the quantity Wi(t) = t — U, where U is the time correspond- 
ing to the last flip of neuron i. Expressed in words, Wi(t) is the number of 
time steps that neuron i stays in the same state, at time step t. Suppose 
that the network is in the cycle of period 2 with maximal amplitude. Then, 
all sites flip in each time step. In this case, Wi = for all site i. Now suppose 
that, after to start, the network quickly arrives to an equilibrium state with 
all the neurons frozen in one state. In this case, after the time step t, all Wi 
are smaller but close to t. Between these two extreme, the distribution P{w) 
gives a good picture of the dynamical state of the network. 

The results for P(w) are shown in figures 3a and 3b. To understand the 
figures, it is sufficient to know that if a neuron had its last flip n time steps 
before t, then it contributes with a count for w = n. The final time we 
adopted is t = 500, since this is a sufficient amount of time to reach the long- 
term behavior. Each figure results from an average over 50 runs. To take 
care of finite size effects, simulations were done for N = 5000 and N = 10000. 
Since the results were equivalent, they are shown only for N = 10000. We 
also have C = 100 and p = 4, so a = 0.04. The initial condition is mo = 0.1. 
Different regions of the phase diagram were visited through changing 9. In 
figure 3a, P(w) is shown for 9 < 1.0. The main feature to be extracted is that 
no freezing occurs: all neurons flip in a time interval smaller than 50 time- 
steps. For 9 = 0.25, the network in the region I* of the phase diagram, inside 
the basin of attraction of the zero phase. The condition 9 = 0.3 corresponds 
to a network in region II*, inside the basin of attraction of the retrieval phase, 
with a retrieval overlap m « 0.1 (see figure 2). Since the values of overlap 
involved are small, we could expect that part of neurons remains frozen, but 
this is in contrast to what actually happens. When 9 = 0.7, the network 
shows chaotic behavior. Since the chaotic attractor wanders through states 
with m going from large negative to large positive values (see figure 2b), we 
could expect, as it is confirmed, that all neurons have flipping activity. For 
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these three values of 9, P(w) shows an exponential decaying behavior, with 
a similar decaying rate. Results for 9 = 0.9 is also included in figure 3a. Now 
we observe a distinct behavior, with P(w) decaying more slowly than in the 
previous cases. We guess that this is already a consequence of the dynamical 
transition that occurs at a slightly higher 9 value, where the chaotic attractor 
splits in two, one for m > and other for m < 0, as can be seen in figure 2b. 

The plot of P(w) for 9 = 1.0, 9 = 1.2 and 9 = 1.3 is shown in figure 3b. 
For 9 = 1.0, the system is in phase V, just above the 9 value where the chaotic 
attractor splits in two. The decaying is slower than exponential, but there 
are not frozen sites. We conclude that the splitting in the chaotic attractor 
is not related to the freezing of a part of sites. Rather, it is related to a 
change in the decaying rate in P(w) that allows for an important decrease 
in the flipping activity. When 9 = 1.2, the network is in the periodic regime 
of phase IV, with positive overlap. The flipping activity decreases, as reveals 
the small population for w ~ 0, and an important concentration of frozen 
sites can be seen for large w. In view of this, we may state that the appearing 
of a periodic retrieval regime with positive m is related to the freezing of part 
of the neurons. It worths to note that P(w) shows a nontrivial structure for 
w > 300. We don't know, at this moment, how to explain this structure. 
Finally, for 9 = 1.3 the system is in retrieval regime. The figure shows a 
small population for small w corresponding to sites that are flipping. There 
is a range for intermediary w with zero or very small population, and a large 
population for large w, corresponding to frozen sites. Since the retrieval 
overlap m « 0.93 is large, we conclude that most of sites are frozen in the 
direction of the pattern. 

4. Conclusion 

Following a dynamical approach, we have studied the extremely diluted non- 
monotonic neural network. The results show that there is no improvement 
in the storage capacity of the retrieval phase due to the non-monotonicity: 
it is always smaller than that of the network with monotonic neurons. As 
stated above, this is in contrast with the highly connected network || . There, 
an optimal 9, whose storage capacity is larger than that of the monotonic 
network, was observed. It would be interesting to study a network with finite 
connectivity, ranging from zero, that is the present model, to 1, that is the 
highly connected model. In this way, it would be possible to localize where 
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is the changeover from one regime to the other. 

Numerical simulations were performed in order to get additional informa- 
tion about the microscopic dynamical evolution. The quantity observed was 
the amount of time that each neuron % is fixed without flip, W{. Excluding 
the trivial cycle-two regime for 9 < 0.2, where P(w) S(w), the distribution 
P(w) shows three distinct regimes, depending on the flipping activity of the 
network: high flipping activity with exponential decaying P(w), mean flip- 
ping activity with less than exponential decay, and low flipping activity with 
freezing. 

We conclude that the distribution P(w) gives a good picture of the mi- 
croscopic evolution of the network, in its different dynamical regimes. It was 
not the aim of this paper, but in a future work it could be interesting to 
investigate quantitatively the decaying law in the second regime. 
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Figure Captions 



Figure 1: Phase diagram for the extremely diluted non-monotonic neural 
network. Solid (dotted) lines correspond to discontinuous (continuous) phase 
transitions. For the dynamical behavior in each phase, see the text. 

Figure 2: Bifurcation diagram and Lyapunov exponent for the extremely 
diluted non-monotonic neural network, for a = 0.04 and varying 9, with the 
initial condition uiq = 0.1. (a) Bifurcation diagram obtained from iteration 
of Eq. (|9]); (b) bifurcation diagram obtained from simulation in a network 
with 10000 neurons, computing the overlap in each time step from its defini- 
tion in Eq. (|8|); (c) Lyapunov exponent, obtained from the iteration of Eq. (j^). 

Figure 3: Distribution P(w) for the extremely diluted non-monotonic neural 
network, for a = 0.04, after simulations with 10000 neurons, (a) threshold 9 
in the range 0.25 - 0.9; (b) threshold 9 in the range 1.0 - 1.3 
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